In prior art, storage management technologies are often used in applications that require writing/reading data to multiple storage media. Storage management is usually done by having a list of registered drives or folders and selecting the appropriate drive to store to.
There are multiple techniques which are used to select the storage location to store data.
U.S. Pat. No. 8,307,176 discloses one technique to register multiple storage locations with a fixed size in a list. Once a storage location is filled up, a next storage location in the list is used. If all storage locations are filled up, all data in the first location in the list is overwritten. For example, in a case where there are three storage locations, Storage A, Storage B and Storage C. When Storage A is filled up, Storage B is used, followed by Storage C. When Storage C is filled up, all data in Storage A is deleted. This may cause some data in Storage A that have not expired to be deleted unnecessarily.
Another technique is to maximize the use of a storage location by performing techniques such as removal of duplicate data and providing a retention policy. The retention policy dictates a time limit a certain file or data in the storage location is retained before it is moved to another media or deleted. In this case, if one or part of the data in the storage location fulfills or exceeds the time limit, the entire storage location is deleted. Data that have not exceeded the time limit are deleted unnecessarily.
In Osisoft PI Server System Management Guide Chapter 3, yet another technique is to apply different data management policies such as a “least recently used” policy. This means that current data is written to a least recently used drive to ensure that all drives are consumed at a fixed rate. Similar to previous prior art described, data in the least recently used drive are deleted before the current data is written. Hence, there may be data that are deleted unnecessarily.
With reference to FIG. 5 of U.S. Pat. No. 7,363,454, a basic flow of a data storage system is illustrated. An application requests free data storage space to store data. The invention of U.S. Pat. No. 7,363,454 then looks at the list of storage locations registered in a storage pool, performs algorithm to select the most appropriate storage location and returns a storage location (e.g. drive volume, folder on the drive volume, etc.) to the requesting application. There appears to be no disclosure on how to handle the scenario when all storage locations are filled.
In a typical industrial plant, large amount of data are stored for many years. In remote industrial environments such as ships or offshore exploration platforms, it is not possible to monitor disk usage daily. Typically, data collection is done in large volume and at higher frequencies, for example, data collection from 2,000,000 field devices in one second intervals. At this rate, 2 million TB of disk space is required each year. The data collected includes process data, alarms and events.
In industrial plants, it is undesirable to use a single hard disk drive to meet system storage requirements and there is a constraint on number of storage media which can be installed due to physical space limitations. A disk full scenario occurs when a disk has reached its storage limit. Currently, when there is a disk full scenario in an industrial plant and there is no available space for new data, the new data is not saved and discarded. This may result in loss of important data. Therefore, disk full scenarios should be avoided as much as possible and solutions are needed to improve the efficiency of handling large data volume.